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The five states in the Regional Educational Laboratory (REL) Northwest Region have 
many rural schools that have been designated as in need of improvement. And 
all five states had rural schools in the first cohort of federal School Improvement 
Grant (SIG) recipients. To address school improvement, the majority of those 
schools implemented the transformation model, which requires strategies related 
to improving instruction, ensuring high-quality staff, and engaging families and 
communities. REL Northwest Region state and district leaders asked REL Northwest 
to conduct a study examining the extent to which rural schools across the nation 
implemented the transformation model, the challenges they experienced, and the 
technical assistance they received. This report provides information about rural 
schools using the transformation model. It is not part of the federal evaluation of 
the SIG program, which provides more comprehensive information about all SIG 
schools. REL Northwest Region leaders may be able to use this study to inform 
future assistance for their rural schools in need of improvement. 
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Summary 


Information about improvement efforts in rural schools is important to state and district 
education leaders in the Regional Educational Laboratory (REL) Northwest Region, which 
includes Alaska, Idaho, Montana, Oregon, and Washington. Not only do these states have 
a large number of rural schools that have been designated as needing improvement, but 
many of those schools received federal School Improvement Grants (SIGs) and used the 
transformation model, which requires strategies related to improving instruction, ensur- 
ing high-quality staff, and engaging families and communities. In the first round of SIG 
funding, almost half (23) of the 49 schools awarded an SIG in the REL Northwest Region 
were rural schools that chose the transformation model for school improvement. Despite 
the large proportion of rural SIG schools using the transformation model, district and state 
leaders believed that a sample of 23 schools was too small to provide the broad information 
necessary for future decisionmaking about rural school improvement in the region. There- 
fore, to obtain meaningful information about rural school improvement efforts, this study 
focused on a national sample of all rural SIG schools implementing the transformational 
model in the first round of funding. 

This study is not part of the federal evaluation of the SIG, which provides more compre- 
hensive information. Leaders participating in research alliances with REL Northwest and 
other regional stakeholders requested this study to learn more about how implementation 
of the SIG transformation model has played out in rural schools across the nation. This 
study will inform their support for this model in rural areas of the REL Northwest Region. 

The transformation approach has evolved over time. In 2009 federal concern about the 
nation’s lowest performing schools led to an unprecedented infusion of SIG funding, as 
well as new requirements for schools receiving those grants (U.S. Department of Educa- 
tion, Office of Elementary and Secondary Education, 2011). The first cohort of SIG schools 
(cohort 1) began implementing the three-year grant programs in the 2010/11 school year. 
Most rural cohort 1 schools (96 percent) chose to implement the transformation model of 
school improvement, one of four possible SIG models (Hurlburt, Le Floch, Therriault, & 
Cole, 2011). As implementation began, education leaders across the country questioned 
whether rural schools would be able to implement some of the requirements, such as those 
involving staff replacement and technical assistance, because of the schools’ geograph- 
ic isolation (Klein, 2010). Leaders in REL Northwest alliances had similar concerns and 
wanted to learn about the extent to which rural schools implemented the transformation 
model, the challenges they experienced, and the technical assistance they received. 

Based in part on concerns about rural schools, the federal government has increased the 
flexibility of the grant requirements. As of the 2015/16 school year, rural schools with new 
SIGs are able to reshape some federal program requirements, including how the trans- 
formation model is implemented (Final Requirements, 2015). The experiences of the first 
cohort of rural SIG schools can inform that process based on what worked, what did not, 
and how schools can do better. This study, drawing information from a national, online 
survey of rural school principals involved in cohort 1 SIG implementation of the transfor- 
mation model, provides preliminary insights for the REL Northwest Region. 

The survey was conducted in spring 2014, after most cohort 1 grant activities were com- 
plete. The survey respondents included 135 principals (67 percent of the 201 schools 


surveyed) in rural schools implementing the transformation model. The most salient find- 
ings include: 

• Few rural schools fully implemented the SIG transformation model. Only 5 percent of 
the principals surveyed said their school had fully implemented the 12 transforma- 
tion strategies that the survey examined. On average, principals said their school 
had fully implemented 6 of the 12 strategies. 

• More schools implemented strategies related to improving instruction than strate - 
gies related to ensuring high-quality staff or engaging families and communities. For 
example, 77 percent of principals reported that their school had fully implemented 
the use of student achievement data to inform instructional decisions, whereas 
52 percent reported that their school had fully implemented staff evaluation 
systems that tied evaluation to student achievement, and 40 percent reported that 
their school had engaged families. 

• More schools reported facing implementation challenges related to ensuring high-quality 
staff and engaging families and communities than challenges related to improving 
instruction. For example, almost half (47 percent) of principals reported challeng- 
es to rewarding staff financially — a strategy related to ensuring high-quality staff 
— and about a third (34 percent) reported challenges to engaging families and 
communities. In contrast, fewer principals (26 percent) reported challenges to 
expanding learning time to improve instruction. 

• Almost all schools received technical assistance from at least one provider, with dis- 
tricts the most frequently identified provider. Most principals (93 percent) reported 
that their school had received technical assistance from at least one provider for 
at least one of the transformation strategies examined in the survey. More prin- 
cipals reported that their school had received this assistance from their district 
(91 percent) than from the state (70 percent), a university (19 percent), or another 
type of organization (42 percent). 

• The more strategies for which principals reported facing challenges, the fewer strategies 
they reported their school had fully implemented. When principals reported challeng- 
es with three or more strategies, they also reported that their school had fully 
implemented an average of only 5.2 strategies. In contrast, when principals report- 
ed challenges with fewer than three strategies, they reported that their school had 
fully implemented an average of 7.5 strategies. 

• The more strategies for which principals reported receiving technical assistance, the 
more strategies they reported that their school had fully implemented. When princi- 
pals reported receiving technical assistance for more than 7 strategies, they also 
reported that their school had fully implemented an average of 7.2 strategies. In 
contrast, when principals reported receiving technical assistance for 7 or fewer 
strategies, they reported that their school had fully implemented an average of 
only 5.7 strategies. 


Contents 


Summary i 

Why this study? 1 

School improvement models 2 

How this study informs rural school improvement efforts 3 

What the study examined 3 

Research questions 4 

Survey participants 4 

What the study found 6 

Few rural schools fully implemented the School Improvement Grant transformation model 6 
More schools implemented strategies related to improving instruction than strategies related 

to ensuring high-quality staff or engaging families and communities 7 

More schools reported implementation challenges related to ensuring high-quality staff and 

engaging families and communities than challenges related to improving instruction 8 

Almost all schools received technical assistance from at least one provider, with districts the 

most frequently identified provider 9 

The more strategies for which principals reported challenges, the fewer strategies they 

reported that their school had fully implemented 9 

The more strategies for which principals reported receiving technical assistance, the more 

strategies they reported that their school had fully implemented 10 

Implications of the study findings 11 

Rural schools are challenged by staffing changes and by engaging families and communities 11 

Rural schools working on improvement need help beyond grant funding 11 

Limitations of the study 12 

Appendix A. Previous studies offer mixed findings A-l 

Appendix B. Survey creation and administration B-l 

Appendix C. Survey instrument C-l 

Appendix D. Analysis methods and additional results for the School Improvement Grant 

rural schools D-l 

Notes Notes-1 

References Ref-1 

Boxes 

1 Previous research on School Improvement Grant strategies in rural schools 3 

2 The survey instrument and analysis methods 5 


Figures 

1 Only 5 percent of rural schools fully implemented all School Improvement Grant 

transformation strategies, and 32 percent partially implemented all strategies in 2014 6 

2 More than two-thirds of principals said that their school had fully implemented 

transformation strategies to improve instruction in 2014 7 

3 More principals reported challenges implementing strategies related to ensuring 

high-quality staff and engaging families and communities than reported challenges 
implementing strategies related to improving instruction in 2014 8 

4 More schools reported that their technical assistance came from districts than from 

the state, universities, or other types of organizations in 2014 9 

5 When principals reported 3 or more strategies with challenges in 2014, they also 

reported that their school had fully implemented an average of 5.2 strategies 10 

6 When principals reported receiving technical assistance for 7 or fewer strategies in 

2014, they reported that their school had fully implemented an average of 5.7 strategies 11 
D1 Schools used a variety of “other” technical assistance providers, 2014 D-5 

Tables 

B1 National Center for Education Statistics classifications for rural areas and towns, 2014 B-3 

B2 Characteristics of schools in the population compared with respondent schools, 2014 B-4 

D1 Percentage of schools reporting no implementation, planning implementation, partial 

implementation, or full implementation of transformation strategies, 2014 D-2 

D2 Percentage of schools reporting that fully implemented transformation strategies were 

essential, 2014 D-2 

D3 Percentage of principals reporting challenges among schools with one or more 

challenges, by type of challenge, 2014 D-3 

D4 Percentage of schools reporting receiving technical assistance, by type of provider, 2014 D-5 

D5 Percentage agreeing or strongly agreeing that technical assistance was sufficient 

and number of respondents, by type of provider, 2014 D-6 

D6 Chi square results for full implementation with at least one challenge, 2014 D-8 

D7 Chi square results for full implementation, by technical assistance, 2014 D-9 


IV 


Why this study? 


Federal School Improvement Grant (SIG) funds for improving low-performing schools 
require actions, such as replacing underperforming staff members, that some rural schools 
have found difficult to implement (Klein, 2010). Concerns about SIG implementation in 
rural areas are particularly important among leaders in the Regional Educational Laborato- 
ry (REL) Northwest Region states — Alaska, Idaho, Montana, Oregon, and Washington — 
where many educators were working to improve rural schools. These schools, by definition, 
were located outside urbanized areas (U.S. Department of Education, National Center for 
Education Statistics, n.d.), far from the resources enjoyed by city and suburban schools, 
such as access to technical assistance providers or a robust work force. 1 

In the REL Northwest Region 23 of the 49 schools participating in the first cohort of 
SIG grants were rural schools using the transformation model, which requires strategies 
related to improving instruction, ensuring high-quality staff, and engaging families and 
communities. All five REL Northwest Region states had at least three rural schools using 
the transformation model. In Montana all SIG schools were rural schools using the trans- 
formation model. 

Although almost half of REL Northwest Region schools in the first SIG cohort were rural 
schools using the transformation model, the number of these schools (23) was small. To 
obtain the broad information necessary for future decisionmaking about rural school 
improvement work in the REL Northwest Region, research alliances collaborating with 
REL Northwest requested this study about the implementation of the transformation 
model in rural SIG schools across the country. The study is not part of the comprehensive 
series of SIG evaluations being conducted by the federal government, which examine a 
larger number of SIG improvement models in a wider variety of schools. 

REL Northwest Region leaders found a limited range of research evidence to guide the 
implementation of the SIG program in rural areas. Although two recent national studies 
showed that SIG schools adopted more federally required improvement strategies than 
similar schools that did not receive the funding, those studies did not disaggregate the 
results by geographic location (Dragoset et al, 2015; Herrmann, Dragoset, & James- 
Burdumy, 2014). Meanwhile, some case studies showed that rural schools and districts 
faced steep challenges in implementing the staff replacement and parental involvement 
practices required by the SIG program (Rosenberg, 2011; Rosenberg, Christianson, Angus, 
& Rosenthal, 2014; Scott, McMurrer, McIntosh, & Dibner, 2012). Despite these studies, 
little was known about how rural schools implemented the strategies of the SIG transfor- 
mation model, the specific challenges they have encountered, or the supports for imple- 
mentation they have received. This study addresses these issues and is particularly relevant 
for the REL Northwest Region, where rural school improvement is a pressing concern for 
the research alliances associated with REL Northwest. 

This study is also important because as of the 2015/16 school year, the U.S. Department of 
Education allows rural SIG grantees to make one modification to the grant requirements, 
as long as that modification meets the original intent of the grant. Additional changes 
allow states to design their own school improvement models, as long as they align with 
the federal intent for School Improvement Grants (Final Requirements, 2015). Districts 
can also choose from models that have improved schools in the past according to at least 
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one study that meets What Works Clearinghouse standards, a set of federal standards that 
ensure studies are of high quality (U.S. Department of Education, Institute of Education 
Sciences, What Works Clearinghouse, 2013). As REL Northwest Region states continue 
to award funds for school improvement, this study provides information about the past use 
of SIG funds in rural areas that state and district leaders can use to inform their modifka- 
tions of the SIG requirements for their rural schools. 

School improvement models 

SIG funding mechanisms have changed over time. Congress put these systems in place as 
part of the No Child Left Behind Act of 2001 (No Child Left Behind Act of 2001, 2002). In 
2009 federal concern about the nation’s lowest performing schools led to an unprecedent' 
ed infusion of funds and guidance for school improvement (U.S. Department of Education, 
Office of Elementary and Secondary Education, 2011). Through the American Recovery 
and Reinvestment Act of 2009, the federal government allocated $3.5 billion for School 
Improvement Grants to support low-performing schools and issued new guidance for the 
grants. Rural schools made up 24 percent of SIG grantees in the first round, awarded in the 
2010/11 school year (cohort 1), and 19 percent of grantees in the second round, awarded in 
the 2011/12 school year (cohort 2; Elurlburt, Therriault, & Le Floch, 2012). 

Federal guidance for School Improvement Grants has also changed. Under the guid- 
ance for the first round of grants, federal funds flowed to states, which were charged with 
identifying eligible low-performing schools and awarding competitive grants to districts 
to improve these schools. The schools were required to implement one of four school 
improvement models (U.S. Department of Education, OESE, 2011): 

• Transformation model, which replaces the principal and requires a number of activ- 
ities to improve the school’s staff, instructional practices, and family and commu- 
nity involvement. 

• Turnaround model, which replaces the principal and 50 percent of the staff, as 
well as requiring operational flexibility needed to implement a variety of school 
improvement activities. 

• Restart model, which closes the school and reopens it as a charter school. 

• Closure model, which closes the school and transfers students to a higher perform- 
ing school. 

Because the transformation model does not require teacher replacement and is less dis- 
ruptive than the other models, some consider it the most flexible of the four (Klein, 2010). 
Perhaps because of this flexibility, it was chosen by about three-quarters of grantee schools 
and 96 percent of rural cohort 1 schools (Elurlburt et al, 2011; Elurlburt et al, 2012). This 
study focuses solely on the transformation model because of its prevalence among rural 
schools. 2 

The transformation model includes specific strategies for school improvement, which 
fall into three broad areas of reform: improving instruction, ensuring the presence of 
high-quality staff, and engaging families and communities. Schools are required to seek 
technical assistance to implement the transformation model from state education agencies, 
local education agencies, or other entities, such as universities or nonprofit organizations. 
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As of the 2015/16 school year, the U.S. Department of Education allows rural schools to 
modify any one strategy of the transformation model (Final Requirements, 2015). Decid- 
ing which strategy to modify may be difficult because research shows no consensus on 
what strategies are effective and practical in rural schools. Some studies have revealed 
challenges in all three areas of the transformation model, as well as in engaging tech- 
nical assistance providers to support implementation (Rosenberg et al, 2014; Rosenberg, 
2011; Sandel & Bhat, 2008; Scott, McMurrer, et al., 2012), while several case studies have 
shown that rural schools can excel in these areas (Barley & Brigham, 2008; Gordon, 2011; 
Hammer, Hughes, McClure, Reeves, & Salgado, 2005; Prater, Bermudez, & Owens, 1997; 
Xu, 2004). 


How this study informs rural school improvement efforts 

In 2009 the SIG program dramatically increased funding to support school improvement. 
Accordingly, it has attracted widespread attention, which has pointed to the need for 
research on its implementation. Have rural schools been able to fully implement the SIG 
transformation model? If not, which strategies have been most challenging? What types of 
technical assistance have increased implementation of the transformation model? Results 
of previous studies are shown in box 1 and appendix A. This study adds to the research 
literature about the implementation of SIGs in rural schools. 

What the study examined 


This study examines implementation of the SIG transformation model of school improve- 
ment in rural schools. It includes descriptive information about the level to which the 
requirements were implemented, the challenges to implementation, and the technical 
assistance supporting implementation. This information was collected through a spring 
2014 survey of principals of cohort 1 rural SIG schools that were using the transformation 
model. These schools typically began their three-year SIG grants in the 2010/11 school 
year. Thus, at the time of the survey, they could be expected to have fully implemented the 
transformation model and the grant would have officially ended (although schools were 
allowed to carry over unspent funds). 
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Box 1. Previous research on School Improvement Grant strategies in rural schools 

Recent case studies have provided preliminary answers to the questions of whether rural 
schools have been able to implement the School Improvement Grant (SIG) transformation 
model, which of its elements are most challenging, and what types of technical assistance 
schools have received. However, none has provided a broad, national overview of SIG imple- 
mentation focused specifically on rural schools. Three studies found that the SIG staffing 
requirements were challenging for rural schools (Rosenberg et al., 2014; Rosenberg, 2011; 
Scott, McMurrer, et al., 2012). One study also found that the family engagement strategies 
were difficult for rural schools (Rosenberg et al., 2014). In contrast, the same study found that 
implementing professional development and increasing time for instruction were among the 
most frequently implemented SIG strategies in rural schools (Rosenberg et al., 2014). Recent 
reports about technical assistance providers in all types of schools suggest that the number 
of nongovernmental providers is growing (Corbett, 2011), but district and state technical assis- 
tance required under School Improvement Grants has been limited (Herman et al., 2013). 
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Research questions 


Survey data were collected to address four research questions: 

• How did principals of rural SIG transformation model schools rate their school’s 
implementation of the model’s requirements? 

• To what extent did principals report challenges to implementation of the transfer- 
mation model? 

• To what extent did principals report that their school received technical assis- 
tance for the implementation of the transformation model? 

• To what extent were principals’ reports of challenges and technical assistance 
related to implementation? 

Survey participants 

Rural SIG transformation schools in cohort 1 represented 42 states and the Bureau of 
Indian Education. These schools served all grade levels and had an average of 460 students. 
On average, 65 percent of their students were a racial/ethnic minority, and 72 percent were 
eligible for the school lunch program, a proxy for few income and a characteristic associat- 
ed with low-performing schools (see table B2 in appendix B). 

Survey respondents included nearly three-quarters (74 percent) of the principals from the 
201 rural SIG cohort 1 transformation schools that were still open and still employed staff 
members who had been employed during the SIG funding period and thus knew enough 
about SIG implementation to respond to the survey. Because 83 percent of schools had 
replaced their principals at some point during the three-year grant period, many principals 
were new. 

Cleaning the data reduced the sample size to 135 principals (67 percent of the 201). Sur- 
veyed principals served schools that were similar in size and student characteristics to the 
total sample. Information about the survey administration and respondents is in appendix 
B; the survey instrument is shown in appendix C. 

The study analyzed relationships between implementation and challenges and between 
implementation and technical assistance. More information about the survey and the 
analysis methods is given in box 2 and appendix D. 
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Box 2. The survey instrument and analysis methods 


Developed and administered by Regional Educational Laboratory Northwest, the spring 2014 survey asked princi- 
pals in rural schools 12 questions about implementation of the School Improvement Grants (SIG) transformation 
model, as described in federal guidance (U.S. Department of Education, OESE, 2011). The following list shows how 
the survey items related to the federal requirements of the transformation model. 

Improving instruction 

• Provide staff with ongoing, high-quality, job-embedded professional development (1 survey item). 

• Use data to identify and implement new research-based curricula (1 survey item). 

• Promote the continuous use of student achievement data in order to inform and differentiate instruction (1 
survey item). 

• Establish schedules and strategies that provide expanded learning time (1 survey item). 

• Use operational flexibility — such as staffing, calendars/time, and budgeting — to improve instruction and student 
outcomes (1 survey item). 

Ensuring the presence of high-quality staff 

• Use rigorous, transparent, and equitable staff and principal evaluation systems that take into account data on 
student growth, as well as other factors (1 survey item). 

• Identify and reward staff, school leaders, and other staff members who improved student outcomes (1 survey 
item) and identify and remove staff members who did not (1 survey item). 

• Implement strategies designed to recruit (1 survey item), place, and retain staff (1 survey item). 

• Replace the principal who led the school prior to commencement of the transformation model (no survey items 
represented this element since responding principals were not present when the original principal was replaced). 

Engaging family and community 

• Provide ongoing mechanisms for family (1 survey item) and community (1 survey item) engagement. 

Technical assistance 

• Ensure that the school receives ongoing, intensive technical assistance and related support from the district, 
the state, or an external organization (the survey included a series of items about district, state, university, and 
other assistance, for previously described strategies). 

To analyze the data, the study team used descriptive statistics for each survey item. It examined the percentages 
of principals who reported that their school had implemented each transformation model strategy, percentages 
who reported challenges to each strategy, and percentages who reported receiving technical assistance for each 
strategy. 

In addition, the study team examined percentages of principals who reported implementation challenges and 
technical assistance across all strategies in the survey. For example, it calculated the percentages of principals who 
said their school had implemented all the strategies, the percentages reporting at least one challenge to implemen- 
tation, and the percentage reporting receiving technical assistance for at least one strategy. 

Finally, the study team explored relationships between implementation and challenges, as well as relationships 
between implementation and technical assistance. 

For more on survey methods, see appendix B; for the survey instrument, see appendix C; and for analysis 
methods, see appendix D. 
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What the study found 


This study confirmed previous research that found certain strategies in the SIG transfer- 
mation model were challenging for rural schools — particularly those related to ensuring 
high-quality staff (Rosenberg et al, 2014; Rosenberg, 2011; Scott, McMurrer, et al, 2012) 
and engaging families (Rosenberg et al, 2014). It also found that schools that had imple- 
mented more strategies reported fewer challenges and more technical assistance. 


The six most salient findings are listed below and described in this section. Additional 
findings are in appendix D. 

• Few rural schools fully implemented the SIG transformation model. 

• More schools implemented strategies related to improving instruction than strat- 
egies related to ensuring high-quality staff or engaging families and communities. 

• More schools reported implementation challenges related to ensuring high-quality 
staff and engaging families and communities than challenges related to improving 
instruction. 

• Almost all schools received technical assistance from at least one provider, with 
districts the most frequently identified provider. 

• The more strategies for which principals reported challenges, the fewer strategies 
they reported that their school had fully implemented. 

• The more strategies for which principals reported receiving technical assistance, 
the more strategies they reported that their school had fully implemented. 
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Few rural schools fully implemented the School Improvement Grant transformation model 

Only 5 percent of rural schools fully implemented all SIG transformation strategies, and 
32 percent partially implemented all strategies (figure 1). Other principals reported that 


Figure 1. Only 5 percent of rural schools fully implemented all School Improvement 
Grant transformation strategies, and 32 percent partially implemented all 
strategies in 2014 
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Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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their school was either still considering how to implement one or more strategies or did not 
intend to implement one or more strategies. At the time of the survey, most of the three- 
year grant activities were expected to be complete, although schools were allowed to roll 
over unused grant funds. 

On average, the 134 principals who responded to survey questions on implementation 
reported that their school had fully implemented at least 6 of the 12 strategies and had 
at least partially implemented 10. Reports ranged from no strategies fully implemented 
(2 percent of principals) to all 12 strategies fully implemented (5 percent of principals). 


More schools implemented strategies related to improving instruction than strategies related to 
ensuring high-quality staff or engaging families and communities 

More than two-thirds of principals said that their school had fully implemented transfor- 
mation strategies to improve instruction, such as using student data to tailor instruction, 
providing professional development, and expanding learning time (figure 2). Strategies for 
ensuring high-quality staff and engaging families and communities were fully implemented 
by a third to a half of schools, according to survey responses. 

If principals reported full implementation of a strategy, they were also asked whether this 
strategy was essential to their school improvement efforts. Almost all reported that the 
fully implemented strategies were essential (see table D2 in appendix D for more details). 
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Figure 2. More than two-thirds of principals said that their school had fully 
implemented transformation strategies to improve instruction in 2014 
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Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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More schools reported implementation challenges related to ensuring high-quality staff and 
engaging families and communities than challenges related to improving instruction 

Principals reported many challenges to implementing the transformation model, including 
insufficient funding, staff expertise, staff time, technology, teacher support, and district 
support (see table D3 in appendix D for the specific challenges principals identified). 

Most principals (78 percent) reported facing at least one challenge to at least one strategy. 
On average, principals reported that their school had experienced implementation chal- 
lenges for three of the strategies they had attempted to implement. 

The percentage of principals reporting challenges varied by transformation strategy 
(figure 3). More principals reported challenges implementing strategies related to ensuring 
high-quality staff and engaging families and communities than reported challenges imple- 
menting strategies related to improving instruction, but at least 20 percent of principals 
reported implementation challenges for each strategy. 


Figure 3. More principals reported challenges implementing strategies related to 
ensuring high-quality staff and engaging families and communities than reported 
challenges implementing strategies related to improving instruction in 2014 
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Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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Almost all schools received technical assistance from at least one provider, with districts the most 
frequently identified provider 


Most schools (93 percent) received technical assistance from at least one provider for at 
least one transformation strategy examined in the survey. The percentage of principals 
reporting that their school had received technical assistance varied across strategies. At 
least one technical assistance provider delivered support for each strategy, but not all 
schools received support for each strategy (see table D4 in appendix D). 


More schools reported that their technical assistance came from districts than from the 
state, universities, or other types of organizations (figure 4). This was true across all strate- 
gies and for each strategy. 

Principals who reported receiving technical assistance were asked the extent to which 
the technical assistance was sufficient. Among the schools receiving technical assistance, 
regardless of the type of service provider, 69-100 percent of principals agreed that it was 
sufficient to help the school implement that aspect of the transformation model (see table 
D5 in appendix D). 

The more strategies for which principals reported challenges, the fewer strategies they reported that 
their school had fully implemented 

Principals identified an average of three transformation strategies that their school found 
challenging to implement. As the number of challenges increased, the number of strategies 
fully implemented decreased (see appendix D). When principals reported 3 or more strategies 
with challenges, they also reported that their school had fully implemented an average of 5.2 


Among the schools 
receiving technical 
assistance, 
regardless of 
the type of 
service provider, 
69-100 percent 
of principals 
agreed that it 
was sufficient to 
help the school 
implement that 
aspect of the 
transformation 
model 


Figure 4. More schools reported that their technical assistance came from districts 
than from the state, universities, or other types of organizations in 2014 
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appendix D for more information. 

Note: n = 135 . 

Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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Figure 5. When principals reported 3 or more strategies with challenges in 2014, they 
also reported that their school had fully implemented an average of 5.2 strategies 
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Source: Authors’ analysis of School Improvement Grant rural principal survey data. 


strategies. In contrast, when principals reported fewer than 3 strategies with challenges, they 
reported that their school had fully implemented an average of 7.5 strategies (figure 5). 

The more strategies for which principals reported receiving technical assistance, the more 
strategies they reported that their school had fully implemented 

Principals reported that their school received technical assistance for an average of seven 
strategies. As the number of strategies for which schools received technical assistance rose, 
the number of strategies fully implemented also rose (see appendix D). When principals 
reported receiving technical assistance for 7 or fewer strategies, they reported that their 
school had fully implemented an average of 5.7 strategies. In contrast, when they reported 
that their school received technical assistance for more than 7 strategies, they reported full 
implementation of an average of 7.2 strategies (figure 6). 

Receiving technical assistance appears to be associated with achieving full implementa- 
tion of strategies. This is also true for many individual strategies. For 7 of the 12 strategies, 
when principals reported receiving technical assistance, they were significantly more likely 
to report full implementation of that strategy. 3 These included: 

• Using operational flexibility (such as staffing, calendars/time, and budgeting) to 
improve student outcomes. 

• Using data to identify and implement a new research-based curriculum. 

• Identifying and removing staff who have not improved student outcomes. 

• Using staff evaluation systems that account for student growth. 

• Implementing strategies to recruit staff who are highly qualified. 

• Identifying and rewarding staff who have improved student outcomes. 

• Providing mechanisms for community engagement. 

(See table D7 in appendix D for more information about this analysis.) 


For 7 of the 12 
strategies, when 
principals reported 
receiving technical 
assistance, they 
were significantly 
more likely 
to report full 
implementation 
of that strategy 
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Figure 6. When principals reported receiving technical assistance for 7 or fewer 
strategies in 2014, they reported that their school had fully implemented an 
average of 5.7 strategies 


Average number of strategies fully implemented 



0-7 strategies with technical assistance 8-12 strategies with technical assistance 

(n = 72) (n = 62) 


Source: Authors' analysis of School Improvement Grant rural principal survey data. 


Implications of the study findings 


The study findings could inform future modifications of the transformation model for 
rural schools in the REL Northwest Region. Using research to guide these modifications 
is important because the federal grants increased flexibility starting in the 2015/16 school 
year. 

Rural schools are challenged by staffing changes and by engaging families and communities 

Education leaders in the REL Northwest Region might want to focus modifications 
on strategies that the survey showed were implemented less frequently. For example, 
77 percent of schools fully implemented the use of student data to inform decisions, while 
only 52 percent fully implemented staff evaluation, and 40 percent fully implemented 
family engagement strategies. Based on these findings, leaders may wish to focus funding 
and attention on the strategies that are more difficult for rural schools to implement. 
However, this study did not examine whether full implementation of these strategies leads 
to improvement in schools, so leaders should monitor school improvement during imple- 
mentation to ensure that these efforts are helpful to schools. 

Rural schools working on improvement need help beyond grant funding 

Across the 12 transformation strategies explored by the survey, the fewer strategies found 
to be challenging, the more strategies that were fully implemented. In addition, the more 
strategies for which technical assistance was reported, the more strategies that were fully 
implemented. Based on these results, state and district leaders in the REL Northwest 
Region may want to consider modifications that alleviate challenges and funnel techni- 
cal assistance to rural SIG schools. However, because this study is descriptive rather than 


State and district 
leaders in the 
REL Northwest 
Region may 
want to consider 
modifications 
that alleviate 
challenges and 
funnel technical 
assistance to 
rural SIG schools 
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experimental, the results cannot determine whether challenges directly inhibited imple- 
mentation or whether the technical assistance directly caused full implementation. There- 
fore, states or other entities need to draw on other data to ensure that the recommended 
technical assistance providers and their support for alleviating challenges have a track 
record of success. 


Ninety-one percent of principals reported that they had received technical assistance from 
their districts, possibly because of districts’ geographic proximity to their rural schools 
or because of the pre-existing relationship between the district and the school. District 
leaders may feel obligated to provide extensive technical assistance, rather than hire 
outside organizations to provide assistance, because of their traditional role supporting 
schools. In addition, districts apply for SIG funding on behalf of their schools. To complete 
these grant applications, districts must have grant-writing capacity. Districts with success- 
ful SIG applications may have more grant-writing resources, and more capacity in general, 
including that needed to provide technical assistance. 

Fewer principals (70 percent) reported receiving assistance from their state. While this 
study cannot determine the optimal level of state assistance to SIG schools, REL North- 
west Region policymakers may want to consider enhancing state supports, particularly for 
strategies for which the survey found that technical assistance was associated with full 
implementation. For example, policymakers and other education leaders in the region 
might provide more state funding or supplement state support with technical assistance 
from other agencies, such as regional education agencies or nonprofit organizations. Federal 
agencies, such as comprehensive centers, content centers, and regional educational labo- 
ratories, might also work with REL Northwest Region states to enhance state supports for 
rural low-performing schools in the future. 


REL Northwest 
Region 

policymakers may 
want to consider 
enhancing 
state supports, 
particularly 
for strategies 
for which the 
survey found 
that technical 
assistance was 
associated with full 
implementation 


Limitations of the study 


This study has several limitations. First, assessment of SIG implementation is based solely 
on principals’ or their proxies’ ratings of their own schools; other informants with knowl- 
edge of the same schools, such as teachers or district or state administrators, might have 
different perspectives. 

Second, principals’ responses may be positively biased by the tendency to provide socially 
acceptable answers on surveys. However, principals showed a range of responses, so the 
confidentiality of the survey may have helped guard against positively skewed results. 

Third, the principal turnover at the schools participating in the study may have had an 
impact on the quality of the responses. However, because principals who were not familiar 
with SIG implementation could designate another administrator or teacher at the school 
to complete the survey, the study used responses from the most knowledgeable person 
available. 

Fourth, 135 principals provided usable surveys, which represented 67 percent of all rural 
SIG transformation schools in cohort 1. The survey respondents may have differed in 
important ways from those who did not respond. However, the study team was able to 
determine that survey respondents came from schools that were similar to the total popu- 
lation of rural SIG transformation schools on important characteristics, including average 
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student enrollment, average percentage of racial/ethnic minority students, and average rate 
of eligibility for the federal school lunch program, an indicator of poverty. 

Fifth, the study provided information only about cohort 1 rural SIG schools implementing 
the transformation model and did not report on other cohorts or other types of improve' 
ment models. 

Finally, the study can make no causal claims concerning how challenges or technical assis- 
tance affected implementation of the transformation model strategies. 
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Appendix A. Previous studies offer mixed findings 


This study grouped the elements of the School Improvement Grants (SIG) transformation 
model into three broad categories: improving instruction, ensuring high-quality staff, and 
engaging family and community. Many education leaders anticipated that rural schools 
would have difficulty meeting some of these requirements (Klein, 2010). Past research has 
shown that rural schools faced challenges in all three areas, but some case studies showed 
that some rural schools excelled in these areas. Similarly, technical assistance to rural 
schools to support these areas has worked in some instances but has been less successful 
in others. This appendix provides an overview of past rural school research in improving 
instruction, ensuring high-quality staff, and engaging family and community, as well as of 
technical assistance supporting these improvement efforts in rural schools. 

Improving instruction 

SIG transformation model strategies for improving instruction include providing pro- 
fessional development, implementing a research-based curriculum, using data to inform 
instruction in the new curriculum, extending the school day, and providing the operation- 
al flexibility to make these things happen. 

Recent case studies showed that these elements were implemented frequently in rural 
schools (Rosenberg et al., 2014). Other research suggests that improving instruction in 
rural areas is possible using innovative approaches, such as online events and mentoring 
(Beesley, 2011) and online learning for teachers (Gordon, 2011). It also shows that success- 
ful professional development training must be tailored to meet the needs of rural schools 
(Howley & Howley, 2005). Finally, while much research shows that adding high-quality 
learning time to the school day improves student outcomes (ECONorthwest & Chalkboard 
Project, 2008; Kidron & Lindsay, 2014), rural schools face multiple challenges to extend- 
ing the instructional day, such as limited availability of private and community partners 
such as tutoring organizations or recreational organizations to assist with before- and after- 
school instruction, long bus rides, and high transportation costs (Sandel & Bhat, 2008). 

Ensuring high-quality staff 

To implement the SIG transformation model, schools must evaluate the effectiveness 
of staff members, remove ineffective staff members, recruit effective staff members, and 
reward and retain effective staff members. Studies of small numbers of rural SIG schools 
found that the staffing elements of the transformation model were particularly difficult to 
implement in rural areas (Rosenberg et al., 2014; Rosenberg, 2011; Scott, McMurrer, et al., 
2012; Yatsko, Lake, Nelson, & Bowen, 2012). 

Several studies found that recruiting and retaining high-quality staff members is more dif- 
ficult in rural areas (Hammer et al., 2005; Monk, 2007) because of lower salaries in rural 
schools (Johnson, Showalter, Klein, & Lester, 2014) and long commute times for teachers 
(Scott, McMurrer, et al., 2012). These challenges may be exacerbated by the fact that small 
rural schools must hire more teachers per 100 students than nonrural schools to cover all 
grade levels and content areas. (Levin, Manship, Chambers, Johnson, and Blankenship, 
2011). 
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Other studies have shown that some rural schools solved their staffing problems through 
partnerships with outside entities, mentoring programs, and distance learning initiatives 
(Barley & Brigham, 2008; Gordon, 2011; Hammer et al., 2005). Universities have offered 
multiple certifications so that teachers in small rural schools can teach multiple grade 
levels, provided online and community college courses to rural teachers and teacher candi- 
dates, and recruited teacher candidates who already live in rural areas (Barley & Brigham, 
2008). In addition, teachers may see the low student-teacher ratio in small rural schools as 
an advantage (Barley & Beesley, 2007). 

Engaging family and community 

Family and community engagement is another SIG transformation model requirement 
that was particularly difficult for rural schools, according to a recent set of case studies of 
nine rural SIG schools (Rosenberg et al., 2014). 

Much of the research literature on family and community involvement in K-12 school set- 
tings has focused on urban and suburban settings (Semke & Sheridan, 2012). Involvement 
includes traditional parent and teacher meetings and community volunteers in schools, as 
well as partnerships between schools and family and community organizations. Rural schools 
experience barriers to this type of involvement according to some research (Semke & Sheri- 
dan, 2012). Other research has shown that rural communities are close knit, that parents visit 
rural schools more frequently than in other locales (Prater et al, 1997), and that they are more 
involved in homework (Xu, 2004). Yet, rural teachers reported that lack of time kept parents 
from becoming involved in schools (McBride, Bae, & Wright, 2002). Also, parents and com- 
munity members in rural areas sometimes mistrust school officials, especially those who come 
from outside the local area (Owens, Richerson, Murphy, Jageleweski, & Rossi, 2007). 

Technical assistance 

To support implementation, SIG guidance mandates that SIG schools receive technical 
assistance from states, districts, or other providers, such as nonprofits or universities. A 
recent study of state and district supports for schools found that it has been limited and 
has varied across states and districts; however, this study did not explore differences in 
support for rural schools (Herman et al., 2013). Similarly, a Government Accountability 
Office report expressed concerns about state and district capacity to monitor the work 
and impact of external technical assistance providers but did not examine rural schools in 
particular (Scott, Sirois, et al., 2012). 

Few large-scale research projects have examined technical assistance for the improvement 
of rural schools, and some of this research is contradictory, perhaps because the results 
may relate to the technical assistance provider. Although regional differences were not 
its primary focus, a study of external supports for low-performing schools in Chicago and 
in urban, suburban, and rural areas of California found that participants viewed the assis- 
tance as peripheral, and rural respondents questioned whether providers understood the 
rural context (Finnigan, Bitter, & O’Day, 2009). In contrast, a study of a regional part- 
nership to provide technical assistance to rural schools in Missouri found that schools 
receiving assistance made progress in adopting a standards-based curriculum, increased 
focused professional development offerings, and improved teacher buy-in to district reform 
activities (Harmon, Gordanier, Henry, & George, 2007). 
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Appendix B. Survey creation and administration 


Regional Educational Laboratory (REL) Northwest created the survey for the 211 cohort 1 
rural School Improvement Grant (SIG) schools implementing the transformation model. 
The survey was based on a review of the literature on SIG grants and rural schools and 
on staff members’ past work on SIG evaluation (Scott, Davis, & Krasnoff, 2012; Scott & 
Lasley, 2013, 2014). The full survey is in appendix C. 

Survey description 

The survey’s introductory section asked principals to state when they became an admin- 
istrator at their school and whether they were familiar with SIG implementation there. 
Principals who were not administrators at the school during the SIG funding period or 
were not knowledgeable about School Improvement Grants were directed to designate a 
proxy. (More information about this process is in the section about collection of informa- 
tion, which follows.) 

The first substantive section of the survey asked principals or their proxies to rate their 
school’s implementation of the required transformation model strategies. 4 These questions 
were based on a similar survey conducted by this study team for the Center on School 
Turnaround (Scott & Lasley, 2013, 2014). 

Principals rated the implementation on a scale of 1 to 4 in which: 

1 = N/A (not intending to implement). 

2 = Planning/developing (planning and preparing to implement this activity). 

3 = Partial implementation (actively implementing this activity, but some aspects are not 

yet completely integrated into routines). 

4 = Full implementation (completely implementing this activity, which is now a routine 

part of our school). 

This scale reflects the stages of implementation developed by Fixsen, Blase, Naoom, & 
Wallace (2009). Finally, this section asked principals who reported full implementation of 
an element whether that element was essential to their school improvement plan. 

An open-ended question allowed principals to list additional school improvement activ- 
ities implemented during the SIG funding period. Flowever, few principals responded to 
this item; thus, the data were not used. 

The next section of the survey gathered information about challenges to implementa- 
tion. For each element, the principals were asked to select one or more challenges. Prin- 
cipals were allowed to select multiple responses because some items could be related to 
one another. For example, “insufficient funding” may mean that staff members were 
overworked and, therefore, also had “insufficient staff time” to implement fully. Principals 
could select both responses if appropriate. They also had the option of selecting “N/A, no 
challenges,” “N/A, no implementation,” or “don’t know.” 
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Challenges included: 

• Insufficient funding. 

• Insufficient staff time. 

• Insufficient district support. 

• Insufficient staff expertise. 

• Insufficient teacher support. 

• Insufficient technological capacity or equipment. 

• Other (list other challenges). 

The challenges section included an open-ended question that allowed principals to elabo- 
rate on the barriers to implementation. 

The last section asked principals to identify the entities that assisted them as they imple- 
mented the required transformation model strategies. Technical assistance entities includ- 
ed their district, their state education agency, a university, or another technical assistance 
provider. This list was based on the entities that other studies found most often assisted 
with School Improvement Grants (Corbett, 2011; Scott, McMurrer, et al, 2012). Principals 
who selected “another technical assistance provider” were prompted to fill in the name. 

Finally, the principal was asked to indicate on a four-point scale the degree to which assis- 
tance from each entity named was sufficient. 

This survey did not ask about one element of the transformation model: removal of the 
principal. The principals who responded to the survey presumably were appointed prin- 
cipal as a result of the SIG award and would not have information about this element or 
challenges to implementing removal of the previous principal. Instead, the survey asked 
principals when they had started at their school. Analysis showed that 83 percent of 
schools did replace their principals at some point during the three-year grant period. 

Definition of rural schools and collection of information 

This one-time data collection by REL Northwest and its partner, Policy Studies Associates, 
solicited responses from the 211 rural SIG schools in cohort 1 implementing the transfor- 
mation model. While there are many definitions of rural schools (Arnold, Biscoe, Farmer, 
Robertson, & Shapley, 2007), this study defined rural schools using the National Center 
for Education Statistics locale codes (table Bl). 

The SIG baseline database created by Flurlburt and colleagues (2012) provided the names 
and addresses of the rural schools receiving School Improvement Grants and using the 
transformation model. The database combines publicly available data from state department 
of education websites, state SIG applications, and the National Center for Education Sta- 
tistics Common Core of Data. The database includes 15,518 SIG-eligible schools across 50 
states, the District of Columbia, and the Bureau of Indian Education. It also designates the 
1,247 SIG-awarded schools across 49 states, including the 211 schools of interest to this study. 

The study included all 211 cohort 1 rural schools across the nation that implemented the 
SIG transformation model — 193 rural schools from the SIG baseline database plus 18 
Bureau of Indian Education schools whose National Center for Education Statistics locale 
codes were missing from the SIG baseline database, but who have locale codes of 31-43 in 
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Table Bl. National Center for Education Statistics classifications for rural areas 

and towns, 2014 

1 Locale code 

Definition I 

31 - Town, Fringe 

Territory inside an urban cluster that is less than or equal to 10 miles from an 

urbanized area. 

32 - Town, Distant 

Territory inside an urban cluster that is more than 10 miles and less than or equal to 

35 miles from an urbanized area. 

33 - Town, Remote 

Territory inside an urban cluster that is more than 35 miles from an urbanized area. 

41 - Rural, Fringe 

Census-defined rural territory that is less than or equal to 5 miles from an urbanized 
area, as well as rural territory that is less than or equal to 2.5 miles from an urban 

cluster. 

42 - Rural, Distant 

Census-defined rural territory that is more than 5 miles but less than or equal to 25 
miles from an urbanized area, as well as rural territory that is more than 2.5 miles 
but less than or equal to 10 miles from an urban cluster. 

43 - Rural, Remote 

Census-defined rural territory that is more than 25 miles from an urbanized area and 

is also more than 10 miles from an urban cluster. 

Source: U.S. Department of Education, National Center for Education Statistics, n.d. 


the National Center for Education Statistics preliminary 2011 data (U.S. Department of 
Education, National Center for Education Statistics, 2013). 

The procedure for collecting information included four steps: a pilot survey, an initial 
invitation letter, the actual survey, and follow-up contacts. Between October 23 and 
November 8, 2013, REL Northwest and Policy Studies Associates conducted a pilot of the 
survey instrument with seven principals from rural SIG schools that had received cohort 1 
funding and had used the transformation model. This pilot allowed analysis of participant 
responses item by item and facilitated revision of the survey to ensure that all items and 
instructions were relevant and easily understood. The pilot also tested the data collec- 
tion procedures. Principals who participated in the pilot had the opportunity to update 
their online pilot survey responses when the actual survey was administered from April 1 
through June 13, 2014. 

After clearance from the federal Office of Management and Budget, data collection began. 
REL Northwest and Policy Studies Associates sent an introductory letter to superinten- 
dents and school turnaround leaders in state departments of education, superintendents of 
districts, and principals of rural SIG transformation schools emphasizing the importance 
of the study and stating that those who responded to the survey would receive a link to 
the published study and would be invited to participate in multiple webinars to discuss the 
study results. The letter also stated that an online survey link would be emailed within a 
week. 

The letter asked recipients who were unfamiliar with SIG at their current schools to fill 
out only the beginning of the survey, which provided instructions for designating a proxy 
(another administrator or teacher leader) to complete the survey (appendix C). The letter 
also explained that principals who were at their school during the time period being 
studied could designate a proxy if it would help the school complete the survey promptly. 

Policy Studies Associates then emailed an invitation to cohort 1 rural SIG transforma- 
tion principals inviting them to complete the online survey via Survey Gizmo, a survey 
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administration software program. Principals were sent two reminders to complete the 
online survey. 

Because return rates were less than 50 percent after the second reminder, gift cards were 
offered to increase the response rate. The pilot had shown that offering an incentive (an 
Amazon electronic gift card) after several reminders increased the response rate from 
44 percent to 77 percent. Several research studies support the use of incentives (Armstrong, 
1975; Church, 1993; James & Bolstein, 1992), particularly for online surveys of principals 
(Jacob & Jacob, 2012). Principals who completed the survey before the second reminder 
were sent the incentive retroactively. After additional reminders, nonrespondents were 
offered a Word document version of the survey via email and U.S. Postal Service mail, in 
case they preferred to respond in one of these formats. 

In the process of emailing and phoning to remind participants to respond to the survey, 
Policy Studies Associates determined that 1 of the 211 schools had closed and 9 had turned 
over their staff completely. This decreased the universe of respondents to 201. Ultimately, 
survey respondents represented a total of 148 (or 74 percent) of the 201 still-intact rural 
SIG transformation schools, (that is, those that were still open and still employed staff 
members who were there during the SIG funding period). Cleaning the data to remove 
duplicates, 5 incomplete responses, and nonresponses resulted in 135 useable surveys 
(67 percent of the target sample). Characteristics of the responding schools were similar to 
those of the total population (table B2). 

Of the 135 respondents, 23 were proxies designated by the current school principal. All 23 
reported that they were familiar with SIG implementation. In addition, 18 of these respon- 
dents indicated that they were a school or district administrator, 2 were teacher leaders, 2 
gave titles that did not clearly indicate whether they were teachers or principals, and 1 did 
not give a title. 


Table B2. Characteristics of schools in the population compared with respondent 
schools, 2014 



Average student 

Racial/ethnic 

Students eligible for 


enrollment 

minority students 

school lunch program 

Schools in population 

Number of 

Number of 

Percent of 

Number of 

Percent of 

Number of 

and survey 

students 

schools 

students 

schools 

students 

schools 

Schools in population (n = 211) 

460 

210 

65 

210 

72 

207 

Schools in survey (n = 135) 

466 

134 

63 

134 

72 

131 


Source: Authors' analysis of School Improvement Grant rural principal survey data and National Center for 
Education Statistics data (2011/12). 
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Appendix C. Survey instrument 


This survey is part of a study of the implementation of School Improvement Grants (SIG) 
in rural schools that use the transformation model. The Regional Educational Laboratory 
(REL) Northwest is conducting this study. REL Northwest is a private, nonprofit research and 
technical assistance provider based in Portland, Oregon, and is part of the Institute of Educa- 
tion Sciences’ Regional Educational Laboratory program (http://ies.ed.gov/ncee/edlabs/). 

The purpose of this study is to provide educators and policymakers with information about 
how rural schools implemented their School Improvement Grants and to inform future 
efforts to turn around schools in rural areas. Your responses are very important. As the 
principal of a SIG school, you have first-hand knowledge about your school’s implementa- 
tion efforts. We will combine your answers with those of other SIG principals to help eval- 
uators understand the implementation of the SIG transformation model in rural schools. 

The survey takes about 20 minutes to complete. We request that you respond by [date to be 
determined]. Because your participation is voluntary, there are no repercussions to partici- 
pating or not participating. By completing this survey, you are consenting to the inclusion 
of your survey data in the REL study. No one, other than the study team members from 
REL Northwest and Policy Studies Associates who are working on this project will see your 
individual responses, and personal information will be removed from the database before 
analysis. Furthermore, the reports prepared for this study will summarize findings across the 
sample and will not associate responses with a specific district, school, or individual. Any 
reported information will always be aggregated across multiple schools. We will not provide 
information that identifies you or your district to anyone outside the study team. 

If you have questions, please contact Dr. Caitlin Scott at caitlin.scott@educationnorthwest. 
org or 1.800.547.6339, ext. 585. 

Thank you for your participation! 

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title 
I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. 
The reports prepared for this study will summarize findings across the sample and will not associ- 
ate responses with a specific district or individual. We will not provide information that identifies 
you or your district to anyone outside the study team, except as required by law. Any willful 
disclosure of such information for non-stadstical purposes, without the informed consent of the 
respondent, is a class E felony. The survey is proprietary and approval by the federal Office of 
Management and Budget (OMB) does not imply endorsement by the Department of Education. 

Public Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are 
required to respond to a collection of information unless such collection displays a valid OMB 
control number. Public reporting burden for this collection of information is estimated to average 
20 minutes, including time for reviewing instructions, searching existing data sources, gathering 
and maintaining the data needed, and completing and reviewing the collection of information. The 
obligation to respond to this collection is voluntary. Send comments regarding the burden estimate 
or any other aspect of this collection of information, including suggestions for reducing this burden, 
to the U.S. Department of Education, 400 Maryland Ave., SWj Washington, DC 20202-4536 or 
email lCDocketMgr@ed.gov and reference the OMB Control Number 1850-0905. 
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Background 


1 . What month did you assume your current position as principal of your school? Please 
check ONE response in the box below the appropriate month. 


January 

February 

March 

April 

May 

June 

July 

August 

September 

October 

November 

December 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


2. What year did you assume your current position as principal of your school? 

Year you became principal at your school: 

3. What position did you hold prior to becoming principal of your current school? 


4- Was this position in the school where you are currently serving as principal? 

□ a. Yes 

□ b. No 

5. Are you familiar with the activities your school implemented using SIG funds, the 
challenges associated with SIG implementation, and the SIG implementation sup- 
ports your school may have received? 

□ a. Yes (Go to question 6) 

□ b. No 

IMPORTANT NOTE: If you are not familiar with implementation of the SIC-funded 
transformation model at your school, we would appreciate it if you would please provide 
the name, title, email address, and phone number of someone whom you believe is familiar 
with SIG implementation at your school, such as: 

• A district administrator responsible for school improvement. 

• A school-level administrator, such as an assistant principal, who was involved in 
the implementation of the SIG-funded transformation model at your school. 

• A teacher who had responsibilities for assisting in the implementation of the 
school’s SIG-funded transformation model. 

Name 


Title 


Email address 


Phone number (including area code) 


[If a principal designated a proxy, the survey ended automatically.] 
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Implementation of the transformation model 


6. To what degree has your school implemented each of the following activities related 
to the SIG transformation model? For each of the following transformation activities, 
please check the first box if your school was not planning to implement that partial- 
lar transformation activity; check the second box if your school is planning and pre- 
paring to implement that activity but has not yet done so; check the third box if your 
school has partially implemented the activity; or check the fourth box if your school 
has fully implemented the activity. Please check only ONE response for each row. 




Implementation stage 


Activity 

N/A 

(not intending 
to implement) 

Planning/ 
developing 
(planning and 
preparing to 
implement this 
activity) 

Partially implemented 
(actively implementing 
this activity, but some 
aspects are not yet 
completely integrated 
into routines) 

Fully implemented 
(completely 
implemented this 
activity, which is 
now a routine part 
of our school) 


Used teacher evaluation systems that account for 
student growth 

□ 

□ 

□ 

□ 

Identified and rewarded teachers who have improved 

student outcomes 

□ 

□ 

□ 

□ 

Identified and removed teachers who have not 

improved student outcomes 

□ 

□ 

□ 

□ 

Provided staff with ongoing, high-quality, 
job-embedded professional development 

□ 

□ 

□ 

□ 

Implemented strategies to recruit high-quality staff 

□ 

□ 

□ 

□ 

Implemented strategies to retain high-quality staff 

□ 

□ 

□ 

□ 

Used data to identify and implement a new 
instructional program 

□ 

□ 

□ 

□ 

Promoted the use of student data to inform instruction 

□ 

□ 

□ 

□ 

Increased learning time for students 

□ 

□ 

□ 

□ 

Provided mechanisms for family engagement 

□ 

□ 

□ 

□ 

Provided mechanisms for community engagement 

□ 

□ 

□ 

□ 

Used operational flexibility (such as staffing, calendars/ 
time, and budgeting) to improve student outcomes 

□ 

□ 

□ 

□ 

Other (Please explain) 

□ 

□ 

□ 

□ 
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7. Among the transformation activities that are fully implemented in your school, which 
of the following, if any, do you believe were essential to your core efforts to improve 
your school? Check ALL that apply. 


Essential to core school improvement efforts? 

Yes, this activity was essential to 

Activity our core school improvement efforts 

No, this activity was not essential to 
our core school improvement efforts 

Using teacher evaluation systems that account for 
student growth 

□ 


□ 

Identifying and rewarding teachers who have improved 

student outcomes 

□ 


□ 

Identifying and removing teachers who have not 
improved student outcomes 

□ 


□ 

Providing staff with ongoing, high-quality, job- 
embedded professional development 

□ 


□ 

Implementing strategies to recruit high-quality staff 

□ 


□ 

Implementing strategies to retain high-quality staff 

□ 


□ 

Using data to identify and implement a new 
instructional program 

□ 


□ 

Promoting the use of student data to inform instruction 

□ 


□ 

Increasing learning time for students 

□ 


□ 

Providing mechanisms for family engagement 

□ 


□ 

Providing mechanisms for community engagement 

□ 


□ 

Using operational flexibility (such as staffing, calendars/ 
time, and budgeting) to improve student outcomes 

□ 


□ 

Other (Please explain) 

□ 


□ 





You are welcome to use the space provided below to elaborate on your responses regarding 
transformation model activities that are fully implemented in your school and that are 


essential to your core efforts to improve your school. 
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8. Did any of the following factors act as barriers to your school’s capacity to implement 
a particular transformation activity? Please indicate whether insufficient funding, staff 
expertise, staff time, technological capacity/equipment, teacher support, or district 
support/guidance acted as a barrier to your school’s capacity to implement each of the 
following transformation activities. Check ALL that apply. 






Challenge 





Activity 

N/A: 

No 

challenge 

Insufficient 
funding to 
implement 
the activity 

Insufficient 

staff 

expertise to 
implement 
the activity 

Insufficient 
staff time to 
implement 
the activity 

Insufficient 
technological 
capacity/ 
equipment 
to implement 
the activity 

Insufficient 
teacher 
support for 
the activity 

Insufficient 
district 
support/ 
guidance for 
the activity 

Don t know/ 
cannot 
specify 
challenge 

N/A: 

Did not 
implement 
this activity 

Using teacher evaluation 
systems that account for 
student growth 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Identifying and rewarding 
teachers who have improved 

student outcomes 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Identifying and removing 

teachers who have not 

improved student outcomes 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Providing staff ongoing, 
high-quality, job-embedded 
professional development 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Implementing strategies to 
recruit high-quality staff 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Implementing strategies to 
retain high-quality staff 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Using data to identify 
and implement a new 
instructional program 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Promoting the use of student 
data to inform instruction 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Increasing learning time for 
students 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Providing mechanisms for 
family engagement 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Providing mechanisms for 
community engagement 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Using operational flexibility 
(such as staffing, calendars/ 
time, and budgeting) to 
improve student outcomes 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Other (Please specify) 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 


You are welcome to use the space provided below to elaborate on your responses regarding 
factors that acted as barriers to your school’s capacity to implement transformation model 
activities. 
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Technical assistance that supports implementation of transformation activities 


9. Please indicate which of the following transformation model activities you received 
assistance for from your school district. Check ALL that apply. 


Activity 

District assistance 
received 

a. 

Using teacher evaluation systems that account for student growth 

□ 

b. 

Identifying and rewarding teachers who have improved student outcomes 

□ 

c. 

Identifying and removing teachers who have not improved student outcomes 

□ 

d. 

Providing staff with ongoing, high-quality, job-embedded professional development 

□ 

e. 

Implementing strategies to recruit high-quality staff 

□ 

t. 

Implementing strategies to retain high-quality staff 

□ 

g. 

Using data to identify and implement a new instructional program 

□ 

h. 

Promoting the use of student data to inform instruction 

□ 

i. 

Increasing learning time for students 

□ 

j- 

Providing mechanisms for family engagement 

□ 

k. 

Providing mechanisms for community engagement 

□ 

1 . 

Using operational flexibility (such as staffing, calendars/time, and budgeting) to 
improve student outcomes 

□ 

m. 

Other (Please specify) 




□ 




10. Please indicate the extent to which you agree that the technical assistance your 
school received from your school district on each of the following SIG transformation 
activities was sufficient. If your school district did not provide technical assistance for 
a particular transformation activity, please select “not applicable” in the appropriate 
row. Check only ONE response for each row. 


The technical assistance provided by my 
school district on this topic was sufficient 


Activity 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

Not 

applicable 

a. Using teacher evaluation systems that account for student growth 

□ 

□ 

□ 

□ 

□ 

b. Identifying and rewarding teachers who have improved student outcomes 

□ 

□ 

□ 

□ 

□ 

c. Identifying and removing teachers who have not improved student outcomes 

□ 

□ 

□ 

□ 

□ 

d. Providing staff with ongoing, high-quality, job-embedded professional 
development 

□ 

□ 

□ 

□ 

□ 

e. Implementing strategies to recruit high-quality staff 

□ 

□ 

□ 

□ 

□ 

f. Implementing strategies to retain high-quality staff 

□ 

□ 

□ 

□ 

□ 

g. Using data to identify and implement a new instructional program 

□ 

□ 

□ 

□ 

□ 

h. Promoting the use of student data to inform instruction 

□ 

□ 

□ 

□ 

□ 

i. Increasing learning time for students 

□ 

□ 

□ 

□ 

□ 

j. Providing mechanisms for family engagement 

□ 

□ 

□ 

□ 

□ 

k. Providing mechanisms for community engagement 

□ 

□ 

□ 

□ 

□ 

1. Using operational flexibility (such as staffing, calendars/time, and 
budgeting) to improve student outcomes 

□ 

□ 

□ 

□ 

□ 

m. Other (Please specify) 







□ 

□ 

□ 

□ 

□ 


You are welcome to use the space provided below to elaborate on your responses regarding 
the sufficiency of the technical assistance your school received from your school district on 
SIG transformation activities. 
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11. Please indicate which of the following transformation model activities you received 
assistance for from your state. Check ALL that apply. 


Activity 

State assistance 
received 

a. 

Using teacher evaluation systems that account for student growth 

□ 

b. 

Identifying and rewarding teachers who have improved student outcomes 

□ 

c. 

Identifying and removing teachers who have not improved student outcomes 

□ 

d. 

Providing staff with ongoing, high-quality, job-embedded professional development 

□ 

e. 

Implementing strategies to recruit high-quality staff 

□ 

t. 

Implementing strategies to retain high-quality staff 

□ 

g. 

Using data to identify and implement a new instructional program 

□ 

h. 

Promoting the use of student data to inform instruction 

□ 

i. 

Increasing learning time for students 

□ 

j- 

Providing mechanisms for family engagement 

□ 

k. 

Providing mechanisms for community engagement 

□ 

1. 

Using operational flexibility (such as staffing, calendars/time, and budgeting) to 
improve student outcomes 

□ 

m. 

Other (Please specify) 




□ 





12. Please indicate the extent to which you agree that the technical assistance your 
school received from your state on each of the following SIG transformation activ- 
ities was sufficient. If your state did not provide technical assistance for a particular 
transformation activity, please select “not applicable” in the appropriate row. Please 
check only ONE response for each row. 




The technical assistance provided 
by my state was sufficient 


Activity 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

Not 

applicable 

a. Using teacher evaluation systems that account for student growth 

□ 

□ 

□ 

□ 

□ 

b. Identifying and rewarding teachers who have improved student outcomes 

□ 

□ 

□ 

□ 

□ 

c. Identifying and removing teachers who have not improved student outcomes 

□ 

□ 

□ 

□ 

□ 

d. Providing staff with ongoing, high-quality, job-embedded professional 
development 

□ 

□ 

□ 

□ 

□ 

e. Implementing strategies to recruit high-quality staff 

□ 

□ 

□ 

□ 

□ 

f. Implementing strategies to retain high-quality staff 

□ 

□ 

□ 

□ 

□ 

g. Using data to identify and implement a new instructional program 

□ 

□ 

□ 

□ 

□ 

h. Promoting the use of student data to inform instruction 

□ 

□ 

□ 

□ 

□ 

i. Increasing learning time for students 

□ 

□ 

□ 

□ 

□ 

j. Providing mechanisms for family engagement 

□ 

□ 

□ 

□ 

□ 

k. Providing mechanisms for community engagement 

□ 

□ 

□ 

□ 

□ 

1. Using operational flexibility (such as staffing, calendars/time, and 
budgeting) to improve student outcomes 

□ 

□ 

□ 

□ 

□ 

m. Other (Please specify) 







□ 

□ 

□ 

□ 

□ 


You are welcome to use the space provided below to elaborate on your responses regarding 
the sufficiency of the technical assistance your school received from your state on SIG 
transformation activities. 
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13. Please indicate which of the following transformation model activities you received 
assistance for from a university. Check ALL that apply. 


Activity 

University 

assistance received 

a. 

Using teacher evaluation systems that account for student growth 

□ 

b. 

Identifying and rewarding teachers who have improved student outcomes 

□ 

c. 

Identifying and removing teachers who have not improved student outcomes 

□ 

d. 

Providing staff with ongoing, high-quality, job-embedded professional development 

□ 

e. 

Implementing strategies to recruit high-quality staff 

□ 

t. 

Implementing strategies to retain high-quality staff 

□ 

g- 

Using data to identify and implement a new instructional program 

□ 

h. 

Promoting the use of student data to inform instruction 

□ 

i. 

Increasing learning time for students 

□ 

j. 

Providing mechanisms for family engagement 

□ 

k. 

Providing mechanisms for community engagement 

□ 

1. 

Using operational flexibility (such as staffing, calendars/time, and budgeting) to 
improve student outcomes 

□ 

m. 

Other (Please specify) 




□ 





14. Please indicate the extent to which you agree that the technical assistance your 
school received from a university on each of the following SIG transformation activ- 
ities was sufficient. If a university did not provide technical assistance for a particular 
transformation activity, please select “not applicable” in the appropriate row. Please 
check only ONE response for each row. 




The technical assistance provided 
by a university was sufficient 


Activity 

Strongly 

disagree 

Disagree 

Agree 

Strongly 

agree 

Not 

applicable 

a. Using teacher evaluation systems that account for student growth 

□ 

□ 

□ 

□ 

□ 

b. Identifying and rewarding teachers who have improved student outcomes 

□ 

□ 

□ 

□ 

□ 

c. Identifying and removing teachers who have not improved student outcomes 

□ 

□ 

□ 

□ 

□ 

d. Providing staff with ongoing, high-quality, job-embedded professional 
development 

□ 

□ 

□ 

□ 

□ 

e. Implementing strategies to recruit high-quality staff 

□ 

□ 

□ 

□ 

□ 

f. Implementing strategies to retain high-quality staff 

□ 

□ 

□ 

□ 

□ 

g. Using data to identify and implement a new instructional program 

□ 

□ 

□ 

□ 

□ 

h. Promoting the use of student data to inform instruction 

□ 

□ 

□ 

□ 

□ 

i. Increasing learning time for students 

□ 

□ 

□ 

□ 

□ 

j. Providing mechanisms for family engagement 

□ 

□ 

□ 

□ 

□ 

k. Providing mechanisms for community engagement 

□ 

□ 

□ 

□ 

□ 

1. Using operational flexibility (such as staffing, calendars/time, and 
budgeting) to improve student outcomes 

□ 

□ 

□ 

□ 

□ 

m. Other (Please specify) 







□ 

□ 

□ 

□ 

□ 


You are welcome to use the space provided below to elaborate on your responses regarding 
the sufficiency of the technical assistance your school received from a university on SIG 
transformation activities. 
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15. Please indicate which of the following transformation model activities you received 
assistance for from another provider. Check ALL that apply. 


Activity 

Assistance 
received from 
another provider 

a. 

Using teacher evaluation systems that account for student growth 

□ 

b. 

Identifying and rewarding teachers who have improved student outcomes 

□ 

c. 

Identifying and removing teachers who have not improved student outcomes 

□ 

d. 

Providing staff with ongoing, high-quality, job-embedded professional development 

□ 

e. 

Implementing strategies to recruit high-quality staff 

□ 

t. 

Implementing strategies to retain high-quality staff 

□ 

g- 

Using data to identify and implement a new instructional program 

□ 

h. 

Promoting the use of student data to inform instruction 

□ 

i. 

Increasing learning time for students 

□ 

j. 

Providing mechanisms for family engagement 

□ 

k. 

Providing mechanisms for community engagement 

□ 

1. 

Using operational flexibility (such as staffing, calendars/time, and budgeting) to 
improve student outcomes 

□ 

m. 

Other (Please specify) 




□ 





16. Please indicate the extent to which you agree that the technical assistance your 
school received from another provider on each of the following SIG transformation 
activities was sufficient. If another provider did not provide technical assistance for 
a particular transformation activity, please select “not applicable” in the appropriate 
row. Please check only ONE response for each row. 



The technical assistance provided by 
another provider was sufficient 


Activity 

Strongly 

disagree 

Disagree 

Agree 

Strongly Not 

agree applicable 

a. Using teacher evaluation systems that account for student growth 

□ 

□ 

□ 

□ 

□ 

b. Identifying and rewarding teachers who have improved student outcomes 

□ 

□ 

□ 

□ 

□ 

c. Identifying and removing teachers who have not improved student outcomes 

□ 

□ 

□ 

□ 

□ 

d. Providing staff with ongoing, high-quality, job-embedded professional 
development 

□ 

□ 

□ 

□ 

□ 

e. Implementing strategies to recruit high-quality staff 

□ 

□ 

□ 

□ 

□ 

f. Implementing strategies to retain high-quality staff 

□ 

□ 

□ 

□ 

□ 

g. Using data to identify and implement a new instructional program 

□ 

□ 

□ 

□ 

□ 

h. Promoting the use of student data to inform instruction 

□ 

□ 

□ 

□ 

□ 

i. Increasing learning time for students 

□ 

□ 

□ 

□ 

□ 

j. Providing mechanisms for family engagement 

□ 

□ 

□ 

□ 

□ 

k. Providing mechanisms for community engagement 

□ 

□ 

□ 

□ 

□ 

1. Using operational flexibility (such as staffing, calendars/time, and 
budgeting) to improve student outcomes 

□ 

□ 

□ 

□ 

□ 

m. Other (Please specify) 







□ 

□ 

□ 

□ 

□ 


You are welcome to use the space provided below to elaborate on your responses regarding 
the sufficiency of the technical assistance your school received from another provider on 
SIG transformation activities. 


Thank you for taking our survey. Your response is very important to us. 
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Appendix D. Analysis methods and additional 
results for the School Improvement Grant rural schools 


This appendix provides detailed information about the data analysis methods for each 
research question, followed by additional results from the analyses for each question. This 
information adds detail to the key results, which are presented in the main report. 

How did principals of rural SIG transformation schools rate their school’s implementation of the 
requirements of the transformation model? 

Analysis methods for research question 1. First, the study team used descriptive statis- 
tics from the 12 items in the survey about implementation of the School Improvement 
Grant (SIG) transformation model. This included frequencies for each item’s four response 
categories. 

Then, the study team recoded responses to these items to create two new series of variables: 

• One in which “1” indicated full implementation and “0” indicated less than full 
implementation, (no implementation, planning, or partial implementation) 

• One in which “1” indicated partial or full implementation and “0” indicated less 
than partial or full implementation, (no implementation or planning) 

Next, the study team summed the resulting variables for full implementation across all 12 
strategies to create a single variable representing the number of strategies (0-12) that each 
school implemented fully. Similarly, the study team summed the variables for partial or 
full implementation to create a single variable representing the number of strategies (0-12) 
that each school implemented at least partially. Descriptive statistics related to these vari- 
ables are included in the results section in the main report. 

The study team also analyzed data related to the series of 12 items asking participants 
whether a fully implemented transformation strategy was sufficient for their school 
improvement efforts. Data were cleaned to ensure that only principals reporting full imple- 
mentation also reported sufficiency. 

Additional results for research question 1 

All 12 survey items representing implementation. Five percent of schools fully implemented 
strategies, and 37 percent at least partially implemented them. Other principals report- 
ed that their school was in the planning stages or was not intending to implement. On 
average, principals said that their school had fully implemented at least 6 of the 12 strate- 
gies and partially implemented at least 10. 

Two percent of principals reported that their school had not even partially implemented 
any of the transformation strategies that the survey examined. These schools may not have 
completed their grants or simply were not able to implement any strategies even partially. 
It is unknown whether any of these schools lost their grants during the grant period. 

Individual survey items. Respondents’ ratings of implementation of individual strategies 
varied (see table D1 and figure 2 in the main report). 
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Table Dl. Percentage of schools reporting no implementation, planning 
implementation, partial implementation, or full implementation of transformation 
strategies, 2014 


Transformation strategy 

Not intending 
to implement 

Planning/ 

developing 

Partially 

implemented 

Fully 

implemented 

Improving instruction 

Data for instruction (n = 133) 

2 

1 

20 

77 

Professional development (n = 134) 

2 

1 

23 

75 

Research-based curricula (n = 134) 

2 

4 

21 

74 

Expanded learning time (n = 134) 

5 

3 

19 

73 

Operational flexibility (n = 132) 

3 

2 

29 

67 

Ensuring high-quality staff 

Staff evaluation (n = 134) 

5 

8 

35 

52 

Staff retention (n = 132) 

8 

11 

41 

40 

Staff rewards (n = 132) 

17 

12 

32 

39 

Staff recruitment (n = 134) 

8 

16 

37 

39 

Staff removal (n = 134) 

16 

19 

34 

31 

Engaging family and community 

Family engagement (n = 134) 

4 

7 

50 

40 

Community engagement (n = 134) 

5 

10 

50 

35 


Note: Rows are ordered within category by percentages of schools reporting full implementation. Percentages 
may not sum to 100 because of rounding. 

Source: Authors' analysis of School Improvement Grant rural principal survey data. 


When principals who reported fully implementing a strategy were asked if that strategy 
was essential to their school’s improvement efforts, 75-98 percent reported that the strat- 
egies were essential (table D2). This survey did not examine whether lack of implementa- 
tion was related to or was caused by principals’ believing some strategies were not essential. 


Table D2. Percentage of schools reporting that fully implemented transformation 

strategies were essential, 2014 


Transformation strategy 

Strategy was essential to 
school improvement efforts 

Improving instruction 

Data for instruction (n = 101) 

98 

Professional development ( n = 97) 

98 

Operational flexibility (n = 85) 

98 

Data-based curricula (n = 96) 

94 

Expanded learning time (n = 95) 

84 

Ensuring high-quality staff 

Staff removal (n = 39) 

97 

Staff retention (n = 49) 

96 

Staff recruitment (n = 49) 

94 

Staff evaluation ( n = 66) 

92 

Staff rewards ( n = 48) 

75 

Engaging family and community 

Family engagement (n = 48) 

83 

Community engagement (n = 42) 

76 

Note: Rows are ordered within category by percentages reporting the strategy was essential. 

Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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To what extent did principals report challenges to implementation of the transformation model? 


Analysis methods for research question 2. Descriptive statistics from the survey questions 
about challenges to implementing the transformation model were used to answer research 
question 2. This included frequencies for the six reported potential challenges to each of 
the SIG transformation strategies that the survey examined. Results are shown by type of 
challenge in table D3. 

The percentage of strategies for which principals reported at least one challenge was cal- 
culated by creating a series of new variables (one variable per transformation strategy) in 
which 1 represented one or more challenges to implementation and 0 represented no chal- 
lenge to implementation. 

To code an element 0, a new variable was calculated to more clearly represent “no chal- 
lenge,” rather than using the survey response “N/A, no challenge.” This was done for two 
reasons. First, a few respondents marked both “N/A, no challenge” and “N/A, did not 
implement this activity.” These responses were removed since respondents in schools that 
did not try to implement would not be able to say whether the activity was challenging. 
Second, a few respondents marked both “N/A, no challenge” and one of the other chal- 
lenges. When this happened, the challenge was counted. Responses from principals who 
responded “don’t know” were removed from the sample. Percentages are shown in figure 3 
in the main report and discussed in more detail in this appendix. 

Additional results for research question 2. Descriptive statistics in the main report 
show the percentages of respondents reporting one or more challenges for each of the 


Table D3. Percentage of principals reporting challenges among schools with one or more challenges, 
by type of challenge, 2014 


Transformation strategy 

Staff 

time 

Funding 

District 

support 

Staff 

expertise 

Teacher 

support 

Technology 

Improving instruction 

Data for instruction (n = 30) 

63 

10 

10 

37 

20 

13 

Professional development (n = 29) 

62 

35 

7 

21 

7 

14 

Data-based curricula (n = 25) 

60 

4 

12 

44 

24 

16 

Expanded learning time (n = 31) 

32 

45 

13 

19 

10 

0 

Operational flexibility (n = 26) 

31 

46 

46 

19 

8 

4 

Ensuring high-quality staff 

Staff evaluation (n = 37) 

38 

19 

30 

24 

22 

14 

Staff removal (n = 39) 

13 

8 

56 

5 

21 

0 

Staff retention (n = 42) 

12 

60 

31 

12 

2 

0 

Staff rewards (n = 52) 

6 

56 

35 

6 

12 

2 

Staff recruitment (n = 46) 

4 

52 

28 

13 

4 

2 

Engaging family and community 

Community engagement (n = 31) 

39 

32 

16 

23 

0 

3 

Family engagement (n = 31) 

36 

32 

13 

32 

0 

3 


Note: Rows are ordered within category by the percentages reporting staff time was a challenge. The highlighted cells represent the 
largest percentage of respondents who reported this challenge. 

Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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transformation strategies (see figure 3). The types of challenges principals reported for each 
of the transformation strategies in the survey are shown in table D3. 

To what extent did principals report that their school received technical assistance to support 
implementation of the transformation model? 

Analysis methods for research question 3. Descriptive statistics from the section of the 
survey that asked about technical assistance were used to answer research question 3. The 
analyses included frequencies for each item that addressed assistance with the implemen- 
tation of the SIG transformation model. The study team also calculated the percentage of 
strategies for which principals reported receiving assistance from at least one provider. To 
do this, the study team created a series of new variables (one variable per transformation 
strategy) in which 1 meant that at least one provider supported implementation and 0 
meant that the school received no technical assistance for that strategy. 

Next, the study team calculated the percentage of schools that received at least some tech' 
nical assistance for at least one of the transformation strategies for each type of provider — 
district, state, university, or other entity. Percentages are reported in figure 4 in the main 
report. 

To further describe the type of technical assistance provider, the study team examined the 
response to open-ended questions that asked the name of any technical assistance provid- 
ers, designated as “other technical assistance providers” for any of the strategies. 

To conduct an exploratory analysis of this qualitative data, the two study team members 
developed codes inductively (Mayring, 2000), meaning that they reviewed all the data 
independently and then agreed on codes. Next, they both coded the data. If they could 
not determine a code for the provider, they searched the Internet to locate information 
about the unknown technical assistance provider. If the provider could not be identified, 
the study team members sought assistance from four REL Northwest staff working on 
school improvement issues. 

Finally, the study team analyzed the data related to the survey items that asked partic- 
ipants to rate the degree to which they believed that the technical assistance from the 
providers was sufficient. Data were cleaned to ensure that only principals reporting that 
their school received technical assistance from a provider also reported on the sufficiency 
of the assistance. 

Additional results for research question 3. The main report describes the technical assis- 
tance providers by type — district, state, university, and other (see figure 4). After district 
and state technical assistance, the most frequent type of provider reported was “other.” 

Respondents who selected “other” were asked to write in the name of the provider. To 
analyze the data from the 48 respondents who wrote provider names, the study team cat- 
egorized the providers as private organization, such as a nonprofit; government organiza- 
tion, such as an educational service district or county office; individual; teachers union; or 
unknown, in the few instances that they did not recognize the name of the provider. Most 
of the “other” providers were private organizations (figure Dl). 


D-4 


Figure Dl. Schools used a variety of “other” technical assistance providers, 2014 


Private organization 
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75 
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Note: n = 48. Percentages total more than 100 percent because some schools used more than one type of 
provider. 

Source: Authors' analysis of School Improvement Grant rural principal survey data. 


The percentage of principals reporting that their school received technical assistance is 
shown in table D4 for each transformation strategy examined in the survey. The percent- 
ages reporting this information by the type of provider — district, state, university, and 
other — are also shown. 


Table D4. Percentage of schools reporting receiving technical assistance, by type 


of provider, 2014 

Transformation strategy 

Any 

Type of technical assistance provider 

technical 

assistance 

District 

State 

University 

Other 3 

Improving instruction 

Professional development 

83 

69 

45 

16 

31 

Data for instruction 

80 

67 

47 

5 

26 

Research-based curricula 

76 

63 

40 

7 

24 

Operational flexibility 

57 

50 

20 

2 

10 

Expanded learning time 

57 

48 

27 

3 

7 

Ensuring high-quality staff 

Staff evaluation 

81 

68 

51 

5 

18 

Staff rewards 

50 

39 

21 

2 

7 

Staff recruitment 

50 

45 

11 

5 

6 

Staff removal 

40 

37 

7 

2 

7 

Staff retention 

39 

39 

0 

0 

0 

Engaging family and community 

Community engagement 

43 

35 

15 

3 

8 

Family engagement 

42 

36 

17 

2 

10 


Note: Rows are ordered within the three categories by the percentage of principals reporting receiving any 
technical assistance, n = 135 for all 12 strategies. 

a. Includes private organizations, government organizations, individuals, teachers unions, or unknown. 
Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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Participants also rated the sufficiency of the technical assistance they received from the 
technical assistance provider, by type: state, district, university, and other. The percentages 
agreeing or strongly agreeing that the assistance was sufficient are shown in table D5 by 
the type of assistance provider. 

To what extent were principals’ reports of challenges and technical assistance related to 
implementation? 

Analysis methods for research question 4 

Challenges and implementation. The relationship between reported challenges and full 
implementation are reported in two ways. First, the variables representing at least one 
challenge were summed to create a new variable representing the number of strategies 
(0-12) for which schools experienced at least one challenge. This variable and the vari- 
able representing the number of strategies fully implemented (0-12) were used to explore 
the relationship between implementation and challenges, using Spearman’s rank-order 
correlation. This nonparametric test, introduced by Spearman in 1904, is similar to the 
Pearson product-moment correlation but can be used on rank-order data or other interval 
data that do not have a normal distribution (Dodge, 2008). Because the two variables of 
interest in this study did not have a normal distribution, a Spearman’s rank-order correla- 
tion was used to examine the relationship between the two. This examination provided 
descriptive information about the relationship between the variables and did not test a 
hypothesis about that relationship. The resulting descriptive statistics are shown in figure 5 
of the main report. 


Table D5. Percentage agreeing or strongly agreeing that technical assistance was sufficient and 
number of respondents, by type of provider, 2014 

Transformation strategy 

District 

State 

University 

Other 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Improving instruction 

Data for Instruction 

90 

89 

92 

62 

97 

34 

100 

6 

Expanded learning time 

89 

64 

81 

36 

100 

10 

100 

3 

Professional development 

89 

92 

83 

60 

93 

42 

95 

21 

Operational flexibility 

88 

66 

84 

25 

85 

13 

100 

2 

Data-based curricula 

87 

83 

92 

53 

88 

33 

100 

9 

Ensuring high-quality staff 

Staff recruitment 

85 

59 

86 

14 

88 

8 

83 

6 

Staff evaluation 

82 

91 

91 

67 

92 

24 

100 

6 

Staff retention 

77 

51 

0 

0 

0 

0 

0 

0 

Staff removal 

72 

49 

88 

8 

89 

9 

100 

1 

Staff rewards 

69 

52 

81 

27 

80 

10 

100 

1 

Engaging family and community 

Community engagement 

83 

46 

84 

19 

82 

11 

100 

3 

Family engagement 

81 

47 

82 

22 

77 

13 

100 

2 

Note: Rows are ordered within category by the percentage agreeing or strongly agreeing district support was sufficient. 


Source: Authors’ analysis of School Improvement Grant rural principal survey data. 
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Second, the study team examined the relationships between challenges and implementa- 
tion, using chi square tests of independence. First introduced by Pearson in the 1900s, the 
chi square test of independence uses a single dependent sample and determines whether 
a statistically significant association exists between variables (Frankie, Ho, & Christie, 
2012). Chi square tests were used descriptively rather than to test a hypothesis about a 
causal relationship between variables. 

For the 12 chi square tests used to examine the relationship between challenges and imple- 
mentation, challenges were represented by the series of dichotomous variables in which 0 
indicated that the respondent had not reported any challenges to implementing the trans- 
formation strategy and 1 indicated that the respondent had reported at least one challenge. 
Full implementation was represented by the series of dichotomous variables in which 0 
indicated that the respondent reported less than full implementation (no implementation, 
planning, or partial implementation) and 1 indicated that the respondent reported full 
implementation. To determine statistical significance, the study team used the Benjamini- 
Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). 

Technical assistance and implementation. Similarly, the study team analyzed the relation- 
ship between technical assistance and implementation in two ways. First, the study team 
summed these variables representing the presence of at least one assistance provider to 
create a new variable representing the number strategies (0-12) for which schools received 
technical assistance. In an analysis similar to the one performed for challenges to imple- 
mentation, the study team used Spearman’s rank-order correlation to explore the rela- 
tionship between the number of strategies (0-12) fully implemented and the number of 
strategies (0-12) supported by technical assistance. As in the previously described analysis, 
the purpose of the analysis was to describe the relationship between the variables rather 
than to test a particular hypothesis. Figure 6 in the main report shows descriptive statistics 
related to the results for this analysis. 

Second, the study team used chi square tests of independence (Frankie, Ho, & Christie, 
2012). Similar to the approach for analyzing the relationships between implementation 
and challenges, a series of chi squares were used to examine the relationship for each of the 
transformation strategies. Technical assistance was represented by series of dichotomous 
variables in which 0 indicated that the respondent had not reported receiving any techni- 
cal assistance related to the strategy and 1 indicated that the respondent had reported at 
least one technical assistance provider supported the strategy. 

As in the previous analyses, full implementation was represented by a series of dichoto- 
mous variables in which 0 indicated the respondent reported less than full implementa- 
tion, (that is, no implementation, planning, or initial implementation) and 1 indicated the 
respondent reported full implementation. To adjust for multiple comparisons, the study 
team used the Benjamini-Hochberg correction to determine statistical significance (Benja- 
mini & Hochberg, 1995). 

Additional results for research question 4 

Challenges. The Spearman correlation showed a statistically significant negative relation- 
ship between challenges and implementation (r 2 = -.32, p < .001). Results of the analysis 
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are described in the main report, and descriptive statistics related to these results are pro- 
vided in figure 5. 

The study team also examined the relationship between challenges and implementation 
for each of the 12 transformation strategies in the survey. Information about the relation- 
ship between challenges to individual elements of transformation and implementation is 
provided in table D6. The association was statistically significant for all 12 strategies exam- 
ined in the survey. 

Technical assistance. The Spearman correlation revealed a statistically significant positive 
relationship between technical assistance and full implementation of strategies (r s 2 = .33, 
p < .001). The association between full implementation and receiving support from one or 
more technical assistance providers is described in the text, and descriptive statistics are 
in figure 6. 

The study team also examined the relationship between technical assistance and imple- 
mentation for each transformation strategy in the survey. Information about this series of 
12 chi square analyses is provided in table D7. The association was statistically significant 
for 7 of the 12 transformation strategies in the survey. 


Table D6. Chi square results for full implementation with at least one challenge, 
2014 



Full implementation with 
at least one challenge 



Transformation strategy 

Actual count 3 

Expected count b 

Number 

Chi square 

Improving instruction 

Expanded learning time 

13 

24.4 

117 

33.80* 

Professional development 

13 

21.2 

124 

17.02* 

Operational flexibility 

11 

17.8 

108 

12.82* 

Data for instruction 

18 

22.9 

123 

6.42* 

Research-based curricula 

14 

18.5 

118 

6.03* 

Ensuring high-quality staff 

Staff rewards 

8 

22.1 

108 

30.36* 

Staff evaluation 

7 

20.2 

117 

27.96* 

Staff retention 

9 

19.5 

99 

18.28* 

Staff recruitment 

10 

20.2 

105 

16.20* 

Staff removal 

7 

13.3 

100 

7.34* 

Engaging family and community 

Family engagement 

6 

14.5 

92 

14.08* 

Community engagement 

8 

13.6 

91 

6.29* 


* Statistically significant after the Benjamini-Hochberg correction. 

Note: Rows are ordered within categories from largest to smallest chi square statistic. Each of the 12 
analyses had 1 degree of freedom. No cells had expected values of less than five. 

a. Number of schools that faced challenges yet fully implemented the strategy. 

b. Number of schools that would have been expected to fully implement this strategy if challenges were not 
associated with implementation. 

Source: Authors’ analysis of School Improvement Grant rural principal survey data. 
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Table D7. Chi square results for full implementation, by technical assistance, 2014 


Transformation strategy 

Full implementation with at least 
one technical assistance provider 

Actual count 3 Expected count b 

Number of 
schools 

Chi square 

Improving instruction 

Operational flexibility 

57 

49.3 

132 

8.14* 

Research-based curricula 

80 

74.6 

134 

6.03* 

Expanded learning time 

61 

55.6 

134 

4.54 

Data for instruction 

85 

82.1 

133 

2.25 

Professional development 

84 

82.8 

134 

0.38 

Ensuring high-quality staff 

Staff removal 

27 

16.2 

134 

17.09* 

Staff evaluation 

65 

56.4 

134 

14.09* 

Staff recruitment 

35 

25.6 

134 

11.08* 

Staff rewards 

34 

25.1 

132 

10.10* 

Staff retention 

25 

20.9 

132 

2.24 

Engaging family and community 

Community engagement 

27 

20.0 

134 

6.58* 

Family engagement 

26 

22.1 

134 

1.90 

* Statistically significant after the Benjamini Hochberg correction. 


Note: Rows are ordered within categories from largest to smallest chi square statistic. Each of the 12 
analyses had one degree of freedom. No cells had expected values of less than five. 

a. Number of schools that had a technical assistance provider and fully implemented the strategy. 

b. Number of schools that would have been expected to fully implement this strategy if technical assistance 
were not associated with implementation. 

Source: Authors' analysis of School Improvement Grant rural principal survey data. 
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Notes 


1. In this study, rural schools are defined as town: fringe, distant, and remote; and rural: 
fringe, distant, and remote (National Center for Education Statistics locale codes 31, 
32, 33, 41, 42, and 43; see table B1 in appendix B). 

2. The SIG baseline database for cohort 1 contains 211 rural schools that chose the 
transformation model and 12 that chose other models. Of the 12 that chose other 
models, 8 chose the turnaround model and 4 chose the restart model. 

3. A series of chi square tests were used to analyze the relationship between techni- 
cal assistance and full implementation. This result was true even after applying the 
Benjamini-Hochberg correction for multiple comparisons. Chi square values ranged 
from 6.03 to 17.09. (See table D7 in appendix D.) 

4. To ensure that each survey item represented only one activity, the study team split 
several activities into multiple survey items. They also eliminated the activity, “Ensure 
that the school receives ongoing, intensive technical assistance and related support 
from the LEA [local education agency], the SEA [state education agency], or an exter- 
nal organization” because this was addressed in a separate section. The SIG require- 
ment to replace the principal was also eliminated because this would have occurred 
before the current principal took that position. 

5. One school had two coprincipals, so the survey was sent to both. Both responded, 
but responses were nearly identical, so the second responding coprincipal’s data were 
eliminated. 
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Studies of cause and effect 


What’s Happening 
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Research methods for educational settings 
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